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Analysis of recently-released NAPLAN tests reveals that the tests cover year levels well 
below the nominal year and that some aspects of the mathematics curriculum are under- 
represented. A high level of untested curriculum content descriptions was found. 


There is considerable public interest in how Australian schoolchildren fare in 
Mathematics. In addition to the normal school reports which are addressed to 
parents/caregivers, the National Assessment Programme: Literacy and Numeracy 
[NAPLAN] (Australian Curriculum Assessment and Reporting Authority [ACARA], n. d. 
a) produces individual student results which are communicated to parents/caregivers as the 
students complete Years 3, 5, 7 and 9. The release of national test results is often 
accompanied in the media by articles with headlines such as “Get back to basics or the 
slide will continue” (Donnelly, 2018). 

This study looks at how well results in the NAPLAN (ACARA, n. d. a) reflect progress 
against mathematics in the Australian Curriculum (ACARA, n. d. b). The NAPLAN may be 
treated in the media as a proxy for the intended curriculum — this is perhaps the ACARA 
brief — but there is no independent research on how close the relationship actually is. 
ACARA is certainly in an interesting, and perhaps conflicted, position as both the author of 
the curriculum and also in charge of producing tests which are supposed to indicate how well 
the curriculum is being taught and learned. In 2018, the NAPLAN test papers from 2010 to 
2016 were publicly released. This event provided the possibility of contemporary and 
independent research into how well the NAPLAN represents the mathematics curriculum. 


Literature Review 


The theoretical basis and motivation of this study are informed by current research on 
the relationship between curriculum and assessment, particularly within mathematics 
education (Cai & Howson, 2013). A curriculum is not simply an official syllabus 
document which prescribes the coverage of topics and performance expectations for the 
learning of mathematics. That way of understanding a curriculum at a system level is better 
known as the “intended” curriculum (Travers & Westbury, 1989). Besides the intended 
curriculum, a curriculum must also be understood and aligned at two other levels — the 
implemented curriculum at a classroom level and the attained curriculum at a student level. 

The conceptualisation of the Australian Curriculum in many ways reflects the goals 
and aspirations of the nation in terms of standardising the formulation of content at the 
implemented curriculum level as well as recommending sound pedagogies at the level of 
the implemented curriculum (Atweh & Goos, 2011). However, there is a gap in terms of 
aligning to the attained curriculum. National testing such as the NAPLAN has the potential 
to assess the attainment of students as a result of the intended and implemented curriculum, 
particularly in mathematics education. Yet, there is little evidence to suggest a strong 
alignment between the intended and attained levels of curriculum, and no research has 
systematically investigated this relationship as manifested in the Australian Curriculum 
and NAPLAN test questions. 


2019. In G. Hine, S. Blackley, & A. Cooke (Eds.). Mathematics Education Research: 
Impacting Practice (Proceedings of the 42"4 annual conference of the Mathematics 
Education Research Group of Australasia) pp. 140-147. Perth: MERGA. 


Currently, the NAPLAN results are widely used as a measure of mathematical attainment 
in Australian schools. For example, the major research project Longitudinal Study of 
Australian Children (MacDonald & Carmichael, 2018) takes the NAPLAN as its yardstick. 


Methodology and Aims 


Practical aspects of Mathematics, and also specialised aspects incorporating the value 
and beauty of Mathematics, should both be incorporated throughout the curriculum notes 
Sullivan, the key author of the current curriculum (2011, p. 5). He details the method by 
which the current curriculum has moved from isolating problem solving to incorporating it, 
along with reasoning and understanding proficiencies, throughout the mathematical 
content. Ability to conduct mathematical operations efficiently is styled “fluency” and also 
included as a proficiency. 

This study focused on Year 5 as indicative of primary NAPLAN questions, and Year 9 
of secondary. Questions from the 2012, 2014 and 2016 NAPLAN papers were coded. For 
Year 9, both the calculator and non-calculator papers were coded. The coding of each 
question was in terms of the most appropriate curriculum Content Description Code (CDC) 
it covers (“classification’’); a difficulty scale (“depth”); a record of any proficiency strands 
to which the question relates (“‘proficiencies”); and an indication if the question linked to 
General Capabilities and Cross Curriculum Priorities. 

The “classification” quantifies the extent to which the curriculum content is mirrored in 
the NAPLAN tests. The “depth” identifies any deficiency in the application of the 
“classification” coverage. The depth coding is on a scale of recall / single-operation 
process / complex process. The “proficiencies” coding identifies which of problem- 
solving, reasoning, understanding and fluency are addressed in each question. 

For example, the 2015 NAPLAN Year 5 Numeracy test includes this item: 


Rhys had 4 fish tanks. Each tank had 10 fish in it. 
He then bought one more tank. 


He shared all the fish equally between the 5 tanks. 
How many fish were then in each tank? 
5 6 
oO © 


© ACARA 2015 
The “classification” of this question is ACMNA076 “Develop efficient mental and written 


strategies and use appropriate digital technologies for multiplication and for division where 
there is no remainder”. The “depth” of this question is complex process. The 
“proficiencies” coding is: problem-solving, reasoning, understanding and fluency. The 
Year 4 Curriculum provides definitions and exemplification of all four proficiencies. 


Year level of the NAPLAN questions 


Associating each question with the most appropriate CDC was time-consuming but 
generally uncontentious. Few questions were difficult to associate with a unique CDC. 
When this was the case, often it occurred because the question covered more than one 
strand of the curriculum, for example the 2012 Year 5 Q20 which asked how many of a 
given irregular shape could be cut from a square grid: both geometrical experimentation 
and calculation are associated. Some questions could be efficiently solved using techniques 
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from later years, such as simultaneous equations or solving quadratics, but perhaps guess- 
and-improve techniques were intended, especially in the calculator-permitted tests. 

NAPLAN tests are held in May each year, therefore they do not focus on content 
from the nominal school year (which are calendar years in Australia), but on content from 
preceding years. It might be expected that the Year 5 tests would mainly be associated with 
Year 4 content, and the Year 9 tests with Year 8 content. To the surprise of the coders, it 
eventuated that codes from much younger year levels often were the most appropriate, as 
shown in Table 1. 


Table 1 School year levels of CDCs in NAPLAN tests of 2012, 2014 and 2016 


Year level of CDC < Year 2 Year 2 Year 3 Year 4 >Year 4 
Year 5 2012 2 5 19 12 2 
Year 5 2014 2 9 8 16 5 
Year 5 2016 1 6 14 17 2 
Overall % 4% 17% 34% 38% 8% 
Year level of CDC < Year 6 Year 6 Year 7 Year 8 >Year 8 
Y9 2012 Non-calculator 9 | (i 5 4 

Y9 2012 Calculator 12 S| 10 2 3 

Y9 2014 Non-calculator 12 7 9 2 2 

Y9 2014 Calculator 6 4 13 ‘i 2 

Y9 2016 Non-calculator 3 4 14 7 4 

Y9 2016 Calculator 3 3 16 8 2 
Overall % 23% 16% 36% 16% 9% 


The questions in the analysed NAPLAN tests map to curriculum year levels 
significantly below the year level of the students taking the tests. This is especially the case 
for Year 9 NAPLAN tests, where 23% of the questions are best associated with CDCs 
from Year 5 and earlier. 


Representation of the Mathematics Strands in NAPLAN Tests 


For each year level, the curriculum breaks down the mathematical content into 
topics within the three strands. Each topic has a CDC and some have further subtopics or 
“elaborations”. As these elaborations were sometimes in the nature of explanations or 
exemplification, they were not considered in the analysis below. 

Year 5 students complete a 40-question NAPLAN test. The proportion of CDCs 
within each strand is compared in Table 2 with the incidence of the Year 4 CDCs in the 
NAPLAN tests of 2012, 2014 and 2016. The number of CDCs in each strand in Years 3, 4 
and 5 is also given. 

Table 2 reveals that in Years 3 and 4 the curriculum applies a heavy weighting to 
Number & Algebra and this is matched by the incidence of Number & Algebra questions 
in the NAPLAN tests. Questions from the Number & Algebra strand comprised about half 
of the 40 questions in each test, matching well with the emphasis given in the curriculum. 
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Table 2 Allocation of CDCs to mathematics content strands in recent Year 5 NAPLAN 


tests 
Number & Measurement & Statistics & 
Algebra Geometry Probability 
CDCs Y3 10 6 4 
CDCs Y4 13 9 6 
CDCs Y5 12 8 5 
NAPLAN Year 5 2012 18 17 5 
NAPLAN Year 5 2014 17 19 4 
NAPLAN Year 5 2016 23 13 4 
NAPLAN Y5 % 48% 41% 12% 
Y4 CDCs % 46% 32% 21% 


As shown in Table 3, the Statistics & Probability strand in the Year 5 curriculum has fewer 
CDCs than the other strands and also significantly less coverage in the NAPLAN tests. Of 
course, it can be argued that in Years 4 and 5 the cognitive content of Statistics & 
Probability is only one-fifth of that in Number & Algebra — and the strand should receive 
only one-fifth of the class time and one-fifth of the weighting in NAPLAN tests. However, 
the Year 4 curriculum does devote six CDCs to Statistics & Probability and 13 to Number 
& Algebra. This suggests that Statistics & Probability should receive a heavier weighting 
in the Year 5 NAPLAN tests. 


Table 3 Allocation of CDCs to mathematics content strands in recent Year 9 NAPLAN 


tests 
Number & Measurement & Statistics & 
Algebra Geometry Probability 
CDCs Y7 19 8 6 
CDCs Y8 12 8 7 
CDCs Y9 10 9 6 
Y9 2012 Non-calculator 11 18 3 
Y9 2012 Calculator 10 19 3 
Y9 2014 Non-calculator 12 18 2 
Y9 2014 Calculator 12 18 2 
Y9 2016 Non-calculator 9 17 6 
Y9 2016 Calculator 9 19 4 
NAPLAN Y9 % 33% 57% 10% 
Y8 CDCs % 44% 30% 26% 


A similar analysis was conducted with the Year 9 figures, also shown in Table 3. 
Year 9 students attempt two NAPLAN tests, one with and one without calculators, each of 
32 questions. The Year 9 NAPLAN tests consistently give far greater importance to the 
Measurement & Geometry strand (57%) than does the curriculum (30%), with 
compensating reductions in the other two strands. 


General Capabilities and Cross Curriculum Priorities in the NAPLAN Tests 


The General Capabilities within the Australian Curriculum are Literacy, Numeracy, 
Information and Communication Technology (ICT) Capability, Critical and Creative 
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Thinking, Personal and Social Capability, Ethical Understanding and Intercultural 
Understanding. The Cross Curriculum Priorities are Aboriginal and Torres Strait Histories 
and Cultures, Asia and Australia’s Engagement with Asia and Sustainability. Very early in 
the process it was recognised that there was little point in coding the General Capabilities 
and Cross Curriculum Priorities aspects of the curriculum. Unsurprisingly, “Numeracy” 
was present in most of the questions and, apart from Literacy and ICT Capability, other 
General Capabilities and Cross Curriculum Priorities were barely represented within 
NAPLAN Mathematics questions. For example, of the 80 questions in the two Year 9 tests 
of 2014 combined, links in the broadest sense are noted in Table 4. No links were found to 
Personal and Social Capability, Ethical Understanding, nor Aboriginal and Torres Strait 
Histories and Cultures. 


Table 4 Links to General Capabilities and Cross Curriculum Priorities in the 2014 Year 9 
NAPLAN tests 


Literacy Numeracy ICT Capability Creative All Others 
Thinking 
et 55 Vy 9 6 


Curriculum Proficiencies in the NAPLAN Questions 


The mathematics curriculum contains four proficiency strands: Understanding (U), 
Fluency (F), Problem Solving (PS) and Reasoning (R). For each of these strands a code of 
“link” or “no link” was given for each question. Table 5 gives totals for all questions asked 
within the two year levels. 


Table 5 Curriculum proficiencies in NAPLAN tests, shown within mathematics strands 
Number & Measurement & Statistics & 
Algebra Geometry Probability 
U F PS R U F PS R U F PS eR 
Year 5 49 39 17 26 28 26 22 20 7 6 0 9 


% 37% 30% 13% 20% | 29% 27% 23% 21% | 32% 22% 0% 41% 
Year9 86 91 25 14 | 59 40 19 £26 | 19 7 2 10 
% 40% 42% 12% 6% | 41% 28% 13% 18% | 50% 18% 5% 26% 


Many links to the Proficiency strands are represented in Table 5, with 
Understanding the most frequent. In Year 9 the proportion of Problem Solving links was 
smaller for Year 5, reflecting a greater emphasis on Understanding. The strands Problem 
Solving and Reasoning may be difficult to assess in a time-constrained situation, but if 
NAPLAN tests are to be taken as a proxy for attainment of the mathematics curriculum 
there needs to be suitable prominence. 


Depth in the NAPLAN Tests 


Each question was coded as Recall, Single-step operation or Complex operation. 
The results, as totals for all questions asked of the year level, are given in Table 6. 


144 


Table 6 Comparison of question depth between Year 5 and Year 9, within strands 


Number & Measurement & Statistics & 
Algebra Geometry Probability 
R S C R S C R S C 
Year 5 5 17 30 16 6 27 4 3 7 
Year 5% 9% 30% 61% | 33% 12% 55% | 29% 21% 50% 
Year 9 16 36 a4 19 6 38 3 8 9 


Year 9% 15% 33% 52% | 30% 10% 60% 15% 40% 45% 


Table 6 reveals that for Year 5 the two strands Measurement & Geometry and 
Statistics & Probability had a far higher percentage of Recall questions than Number & 
Algebra. In Year 9, Recall questions occurred at twice the rate in Measurement and 
Geometry compared with the other two strands. In both Year 5 and Year 9, a high 
proportion of the Measurement & Geometry questions were Recall. This may indicate a 
propensity to favour activities such as naming, and to avoid applications of Measurement 
& Geometry in the NAPLAN tests. It is already noted that the Year 9 NAPLAN tests give 
greater importance to Measurement & Geometry questions, and this section reveals that the 
Measurement & Geometry questions tend to favour Recall over more complex intellectual 
demands. 


Comparison of Calculator Permitted or not in Y9 NAPLAN Questions 


Six Year 9 NAPLAN tests were coded, three where calculators were expected to be 
available and three where calculators were not permitted. The distribution of questions 
across the three content strands is uneven, hence analysis was made within each strand. 

For Number & Algebra, questions of greater depth demands are provided when 
calculators are available (63%) compared with non-calculator questions (43%). A higher 
number of operations is required to solve the questions employed in tests where calculators 
are available, except in Statistics & Probability. However, there was little difference in the 
questions in terms of the curriculum proficiencies addressed. The tests where candidates 
had access to calculators provided similar levels of focus on problem solving and 
reasoning as the tests where no calculator use was permitted. Without calculator access, 
these were 14%, 33% and 39% for Number & Algebra, Measurement & Geometry and 
Statistics & Probability respectively. With calculators allowed, the figures were 22%, 19% 
and 20% respectively. 


Curriculum Content Descriptions in the NAPLAN Questions 


By combining three years of NAPLAN tests, an analysis was made of the incidence 
of CDCs which the questions addressed. Many CDCs escaped having questions which 
could be ascribed to them, while other CDCs proved popular. The graphs below show the 
results of combining the 120 questions of the three Year 5 NAPLAN tests which were 
coded and the 192 Year 9 questions. The horizontal axis is the incidence of each of the 
CDCs. The outlier which occurred seven times in Year 5 is ACMNA082: “Solve word 
problems by using number sentences involving multiplication or division where there is 
no remainder’. The Year 9 outlier is ACMNA188: “Solve a range of problems involving 
rates and ratios, with and without digital technologies”. 
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Year 5 NAPLAN tests 2012, Year 9 NAPLAN tests 2012, 


2014 & 2016 2014 & 2016 
20 40 
15 30 
10 20 
5 10 
0 al 2 3 4 5 6 7 0 1 2 3 4 #5 6 7 8 


A notable feature of these graphs is the large number of CDCs which did not 
appear in the three years of NAPLAN tests analysed. In the mathematics curriculum for 
Years 3 and 4 there are 48 CDCs, nine of which were not represented in the three Year 5 
tests. These include ACMNAO77 on equivalent fractions used in contexts and ACMSO094 
on events where the chance of one will not be affected by the occurrence of the other. In 
the mathematics curriculum for Years 6, 7 and 8 there are 88 CDCs. In the six Year 9 tests, 
31 of these CDCs were not represented. These include ACMNA182 on the index laws, 
ACMMG196 on perimeters and areas of quadrilaterals such as trapeziums and ACMSP293 
on variation of statistics in random samples drawn from the same population. 

By frequency of CDCs, in three years of NAPLAN tests 19% of the mathematics 
topics Year 5 students should be able to do was not tested and 35% of the mathematics 
topics Year 9 students should be able to do was not tested. Only 23% of the Year 5 
questions and 28% of the Year 9 questions were not multi-choice, but only numerical 
responses were permitted anyway. This inhibits testing of some topics, such as the ability 
to draw a graph or to explain a plan for drawing a random sample. Of course, some CDCs 
from different years have similar applications, some CDCs may be difficult to cover ina 
test format, or there may be other explanations, but the level of untested intended 
curriculum is significant. 


Conclusions 


A significant proportion of the intended mathematics curriculum topics were not 
covered in the three years of NAPLAN tests analysed. Those topics covered are often at a 
much younger age level than the school year level of the students. The curriculum 
proficiency strands of Problem Solving and Reasoning have very limited coverage, as may 
be expected in what are mainly multi-choice questions. 


Limitations and Suggestions for Further Research 


Care was taken in the coding of questions, but this is not a task which can be done 
with complete objectivity and reliability. Partitioning a curriculum into discrete CDCs is 
unlikely to result in equal weightings for each content description. This study therefore 
reports only on differences which are quite large in scale. In terms of readability, 
appropriateness and proportionate representation of the curriculum it would be interesting 
to compare the NAPLAN questions with questions employed in textbooks and in online 
teaching resources. 

Data for this study was obtained from the most recently released NAPLAN test 
questions, up to 2016. Subsequent tests have not been publicly released so that a Rasch 
question-selection system can be implemented in online tests. Future auditing of NAPLAN 
online tests may be more difficult as they will employ large banks of test items. It is a 
major change that the NAPLAN can be sat online and many opportunities for research 
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present themselves. Students may regard computer-hosted tests as having less importance 
than a booklet where their own name is printed on the cover. Without something to write 
on, students may approach questions differently, perhaps being more likely to guess 
answers. The ACARA website contains a link to a report commissioned from market 
research agency Colmar Brunton (2019) which relies on parent perception of the 
difficulties students faced in the trial online tests, but a more direct appreciation is possible. 
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